Max Margin Dimensionality Reduction

A fundamental problem in machine learning is to extract compact but relevant representations of empirical data. Relevance can be measured by the ability to make good decisions based on the representations, for example in terms of classification accuracy. Compact representations can lead to more human-interpretable models, as well as improve scalability. Furthermore, in multi-class and multi-task problems, learning a unified input representation that is shared among classes or tasks can reduce the number of free parameters and sample-complexity. Sharing representation is also a powerful method for transfer learning [4, 11], where the representation learned from some tasks is used to facilitate learning other tasks. The naive but common practice for finding a compact representation is an unsupervised pre-processing phase (like clustering or PCA) followed by classification in the reduced space. A preferable approach is to learn the features simultaneously with the classifier. Several methods approached this problem, including the well known Fisher’s Linear Discriminant Analysis (LDA), and its variants [7], learning metrics for kNN [9, 8], and extracting features for multi-task classification [3, 2]. Here we focus on finding low-dimensional linear projections that are optimized for support vector machines, in a singleor multitask setting. Formaly, we have k = 1, . . . ,K binary classification tasks, each with labeled data xkik ∈ R , y ik ∈ {−1,+1}, ∀ik = 1, . . . , nk, we look for a rank-d linear projection A, d ≤ D, that is shared across all tasks, and K classifiers w, one for each task. We optimize jointly over all the classifiers w = {